<!DOCTYPE html>
<!-- saved from url=(0014)about:internet -->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<meta http-equiv="x-ua-compatible" content="IE=9" >

<title>Chapter 3, Problem 7</title>

<style type="text/css">
body, td {
   font-family: sans-serif;
   background-color: white;
   font-size: 12px;
   margin: 8px;
}

tt, code, pre {
   font-family: 'DejaVu Sans Mono', 'Droid Sans Mono', 'Lucida Console', Consolas, Monaco, monospace;
}

h1 { 
   font-size:2.2em; 
}

h2 { 
   font-size:1.8em; 
}

h3 { 
   font-size:1.4em; 
}

h4 { 
   font-size:1.0em; 
}

h5 { 
   font-size:0.9em; 
}

h6 { 
   font-size:0.8em; 
}

a:visited {
   color: rgb(50%, 0%, 50%);
}

pre {	
   margin-top: 0;
   max-width: 95%;
   border: 1px solid #ccc;
   white-space: pre-wrap;
}

pre code {
   display: block; padding: 0.5em;
}

code.r, code.cpp {
   background-color: #F8F8F8;
}

table, td, th {
  border: none;
}

blockquote {
   color:#666666;
   margin:0;
   padding-left: 1em;
   border-left: 0.5em #EEE solid;
}

hr {
   height: 0px;
   border-bottom: none;
   border-top-width: thin;
   border-top-style: dotted;
   border-top-color: #999999;
}

@media print {
   * { 
      background: transparent !important; 
      color: black !important; 
      filter:none !important; 
      -ms-filter: none !important; 
   }

   body { 
      font-size:12pt; 
      max-width:100%; 
   }
       
   a, a:visited { 
      text-decoration: underline; 
   }

   hr { 
      visibility: hidden;
      page-break-before: always;
   }

   pre, blockquote { 
      padding-right: 1em; 
      page-break-inside: avoid; 
   }

   tr, img { 
      page-break-inside: avoid; 
   }

   img { 
      max-width: 100% !important; 
   }

   @page :left { 
      margin: 15mm 20mm 15mm 10mm; 
   }
     
   @page :right { 
      margin: 15mm 10mm 15mm 20mm; 
   }

   p, h2, h3 { 
      orphans: 3; widows: 3; 
   }

   h2, h3 { 
      page-break-after: avoid; 
   }
}

</style>



<!-- MathJax scripts -->
<script type="text/javascript" src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>



</head>

<body>
<h1>Chapter 3, Problem 7</h1>

<h2>Generic Solution</h2>

<p><strong>Proposition</strong>: Prove that in case of simple linear regression:</p>

<p>\[  y = \beta_0 + \beta_1 x + \varepsilon  \]</p>

<p>the \( R^2 \) is equal to correlation between X and Y squared, e.g.:</p>

<p>\[  R^2 = corr^2(x, y)  \]</p>

<p>We&#39;ll be using the following definitions to prove the above proposition.</p>

<p><strong>Def</strong>:
\[  R^2 = \frac{TSS - RSS}{TSS}  \]</p>

<p><strong>Def</strong>:
\[  TSS = \sum (y_i - \bar{y})^2 \label{TSS}  \]</p>

<p><strong>Def</strong>:
\[  RSS = \sum (y_i - \hat{y}_i)^2 \label{RSS}  \] </p>

<p><strong>Def</strong>:
\[ 
\begin{align}
  corr(x, y) &= \frac{\sum (x_i - \bar{x}) (y_i - \bar{y})}
                     {\sigma_x \sigma_y} \\
  \sigma_x^2 &= \sum (x_i - \bar{x})^2 \\
  \sigma_y^2 &= \sum (y_i - \bar{y})^2
\end{align}
 \]</p>

<p><strong>Proof</strong>:</p>

<p>Substitute defintions of TSS and RSS into \( R^2 \):</p>

<p>\[ 
R^2 = \frac{\sum (y_i - \bar{y})^2 - \sum (y_i - \hat{y}_i)^2}
           {\sum (y_i - \bar{y})^2}
 \]</p>

<p>Let&#39;s work on the numerator:</p>

<p>\[ 
\begin{align}
  A &= \sum (y_i - \bar{y})^2 - \sum (y_i - \hat{y}_i)^2 \\
    &= \sum \left[ (y_i - \bar{y}) - (y_i - \hat{y}_i) \right] 
            \left[ (y_i - \bar{y}) + (y_i - \hat{y}_i) \right] \\
    &= \sum (\hat{y}_i - \bar{y})
            (2y_i - \bar{y} - \hat{y}_i)
\end{align}
 \]</p>

<p>Recall that:</p>

<p>\[ 
\begin{align}
  \hat{\beta}_0 &= \bar{y} - \hat{\beta}_1 \bar{x} \label{beta0} \\
  \hat{\beta}_1 &= \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}
                        {\sum (x_j - \bar{x})^2}
\end{align}
 \]</p>

<p>Substitute the expression for \( \hat{\beta}_0 \) into \( \hat{y}_i \):</p>

<p>\[ 
\begin{align}
  \hat{y}_i &= \hat{\beta}_0 + \hat{\beta}_1 x_i \\
            &= \bar{y} - \hat{\beta}_1 \bar{x} + \hat{\beta}_1 x_i \\
            &= \bar{y} + \hat{\beta}_1 (x_i - \bar{x})
\end{align}
 \]</p>

<p>Let&#39;s analyze two terms from \( A \):</p>

<p>\[ 
\begin{align}
         \hat{y}_i - \bar{y} &= \hat{\beta}_1 (x_i - \bar{x}) \\
  2y_i - \bar{y} - \hat{y}_i &= 2y_i - \bar{y} - \bar{y} -
                                \hat{\beta}_1 (x_i - \bar{x}) \\
                             &= 2(y_i - \bar{y}) - 
                                \hat{\beta}_1 (x_i - \bar{x}) 
\end{align}
 \]</p>

<p>and substitute these expressions back into \( A \):</p>

<p>\[ 
\begin{align}
  A &= \sum \hat{\beta}_1 (x_i - \hat{x})
            \left[ 2(y_i - \bar{y}) - \hat{\beta}_1 (x_i - \bar{x}) \right] \\
    &= \hat{\beta}_1 \sum (x_i - \bar{x})
                          \left[ 2(y_i - \bar{y}) -
                                 \hat{\beta}_1 (x_i - \bar{x}) \right] \\
    &= \hat{\beta}_1
       \left[ 2 \sum (x_i - \bar{x})(y_i - \bar{y}) -
              \hat{\beta}_1 \sum (x_i - \bar{x})^2 \label{A4} \right]
\end{align}
 \]</p>

<p>Using formula for \( \hat{\beta}_1 \) it is easy to see that the last term is
nothing but:</p>

<p>\[  \sum (x_i - \bar{x}) (y_i - \bar{y})  \]</p>

<p>Thus, we get:</p>

<p>\[ 
\begin{align}
  A &= \hat{\beta}_1 \sum (x_i - \bar{x}) (y_i - \bar{y}) \\
    &= \frac{\left[ \sum (x_i - \bar{x}) (y_i - \bar{y}) \right]^2}
            {\sum (x_j - \bar{x})^2}
\end{align}
 \]</p>

<p>Plug the final expression for \( A \) back into \( R^2 \):</p>

<p>\[ 
R^2 = \frac{\left[ \sum (x_i - \bar{x}) (y_i - \bar{y}) \right]^2}
           {\sum (x_j - \bar{x})^2 \sum (y_k - \bar{y})^2}
 \]</p>

<p>Compare this to the definition of correlation and get:</p>

<p>\[  R^2 = corr^2(x, y)  \]</p>

</body>

</html>

